A systematic review of whole disease models for informing healthcare resource allocation decisions

Background Whole disease models (WDM) are large-scale, system-level models which can evaluate multiple decision questions across an entire care pathway. Whilst this type of model can offer several advantages as a platform for undertaking economic analyses, the availability and quality of existing WDMs is unknown. Objectives This systematic review aimed to identify existing WDMs to explore which disease areas they cover, to critically assess the quality of these models and provide recommendations for future research. Methods An electronic search was performed on multiple databases (MEDLINE, EMBASE, the NHS Economic Evaluation Database and the Health Technology Assessment database) on 23rd July 2023. Two independent reviewers selected studies for inclusion. Study quality was assessed using the National Institute for Health and Care Excellence (NICE) appraisal checklist for economic evaluations. Model characteristics were descriptively summarised. Results Forty-four WDMs were identified, of which thirty-two were developed after 2010. The main disease areas covered by existing WDMs are heart disease, cancer, acquired immune deficiency syndrome and metabolic disease. The quality of included WDMs is generally low. Common limitations included failure to consider the harms and costs of adverse events (AEs) of interventions, lack of probabilistic sensitivity analysis (PSA) and poor reporting. Conclusions There has been an increase in the number of WDMs since 2010. However, their quality is generally low which means they may require significant modification before they could be re-used, such as modelling AEs of interventions and incorporation of PSA. Sufficient details of the WDMs need to be reported to allow future reuse/adaptation.


Introduction
Health economic models are routinely used to inform healthcare resource allocation decisions in many countries across the world [1][2][3][4][5].Models provide an explicit means of structuring a decision problem and synthesising all relevant evidence to estimate the expected costs and consequences of alternative health care interventions within a given health condition, usually over a lifetime horizon.Conventional health economic models are 'piecewise' in that they typically address a single decision problem at a specific decision point in a care pathway.'Piecewise' models represent the standard analytic approach for informing decisions about the availability of health technologies by the National Institute for Health and Care Excellence (NICE) and similar agencies elsewhere [6][7][8], but they are subject to several limitations [9].The first of these relates to the failure to capture system interdependencies between different interventions.The cost-effectiveness of any new intervention is dependent not only on the costs and effectiveness of the new intervention itself, but also on the configuration of the prevailing system, i.e. the availability, costs and effectiveness of existing interventions [9,10].For example, the cost-effectiveness of a new test for a given cancer type may be dependent on currently recommended treatment options for patients with diagnosed disease, as well as the availability of a screening programme for asymptomatic individuals.This type of system-level interdependency between interventions used for the same condition is seldom adequately captured by piecewise models due to their limited scope.Second, piecewise models often employ a simple piecewise cost per quality-adjusted life year (QALY) threshold rule which does not explicitly consider the budget constraint [11,12].However, it has been well documented that the repeated application of a threshold-based decision rule may lead to uncontrolled growth in health-care expenditure [13][14][15][16][17][18].Third, most models are developed with the intention of informing a single decision problem within a broader pathway of care.This means that across a whole disease area, reimbursement and coverage decisions are based on a number of asynchronously developed discrete economic models which tend to apply different model structures, assumptions and evidence.This can lead to a situation whereby two models addressing the same decision question produce inconsistent conclusions, with potential to lead to suboptimal adoption decisions [19][20][21][22][23][24][25].
System-level models, which include important events, health outcomes and costs across an entire disease area, represent a potential means of addressing the limitations of conventional piecewise models.Three well-known examples of system-level models include the US Archimedes diabetes model [26], the US Coronary Heart Disease (CHD) Policy Model [27], and the UK CHD model [28].Although this type of modelling approach dates back to 1977 [29], it was not well-defined until 2012 when Tappenden et al. set out a methodological framework for whole disease models (WDM) [9].In short, a WDM is a system-level generic disease model which allows for the consistent economic analysis of options across entire disease and treatment pathways, including prevention, detection, diagnosis and treatment [9].Owing to the broader scope of these models, which focus on the whole disease and treatment pathway rather than individual decisions within that pathway, WDMs can provide a consistent conceptual and mathematical platform for the economic analysis of a large number of health care interventions based on a single model.In addition, WDMs can allow for the consideration of a variety of different economic decision rules which jointly deal with investment and disinvestment decisions through reference to a budget constraint, such as the disease-level constrained maximisation of health decision rule [30,31].Under this decision rule, different combinations of healthcare services (investment and disinvestment options) are tested [9].The one that maximises health benefits while staying within the available budget is considered the most costeffective choice.
Despite the potential benefits of the approach, the development of a WDM requires a significant initial investment of time and resources [9,32] and presents additional challenges for model verification and validation.The initial investment in WDMs is therefore of greater value if they are re-used and adapted over time.However, re-use requires other modellers to be aware of existing WDMs and to determine their quality, as this will impact on whether the WDMs can be re-used.To our knowledge, no previous studies have systematically reviewed which WDMs exist, or the extent to which previously developed models could be re-used to address future decisions.To fill this gap, this systematic review aims to identify existing models meeting the criteria of a WDM for any disease (regardless of whether they were labelled as a WDM), to critically examine the quality of these models and to provide recommendations to improve the quality, reporting, and adaptability for future WDMs.

Methods
This systematic review was conducted according to the PRISMA recommendations for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions [33].The protocol of this review was registered with PROSPERO (https://www.crd.york.ac.uk/ prospero/) on 12 th October 2020 (CRD42020199875).

Inclusion/exclusion criteria
Inclusion and exclusion criteria were defined a priori.It was hypothesised that models which strictly meet the criteria of a WDM defined by Tappenden et al [9] may be rare.Therefore, for the purpose of this systematic review, we decided to include studies which report models that broadly meet the criteria of being a WDM, i.e. a model which can evaluate multiple decision points (i.e.�3) for people with a given disease, and people at risk who may, or may not, go on to develop the health condition, and thus evaluates both the prevention and treatment of the condition simultaneously.Throughout this paper, these are referred to as 'WDMs', regardless of whether they were reported as a WDM in the original study.Those narrower models which evaluate multiple decision points (i.e., �3) only for people with a given disease (thus excluding people at risk of the disease), or only for people at risk of the disease (and thus excluding people with the given disease), are referred to as 'pathways models'.These pathways models were excluded from the review as they do not meet our inclusion criteria of a WDM; however, a brief summary of identified pathways models is provided.
A model was assessed as meeting the criteria of a WDM either by demonstration (i.e., the authors used the model to address three or more decision points and reported the results in the paper) or based on authors' reporting (i.e., the authors did not use the model to address evaluate options at multiple decision points in the paper, but they clearly reported that the model can be used in such a way).No limits were applied to the searches or the review inclusion criteria regarding specific diseases or conditions under consideration, types of economic evaluation, population, intervention or comparator, or outcome measures.Only published papers were included.Studies were excluded if they met any of the following criteria: (i) reviews, commentaries, letters, editorials, or abstracts; or (ii) not reported in English.
Whether a paper meets our inclusion criteria or not was determined based on the content of each individual paper, rather than the content of a series of related papers.Therefore, models which were used to address multiple decision points in a series of papers, but each paper only used that model to address one or two decision points were excluded.

Search strategy
Electronic biomedical databases searched included MEDLINE (including in-Process & other non-indexed) and EMBASE which were searched through the Ovid interface (https://ovidsp.ovid.com/).In addition, the NHS Economic Evaluation Database (NHS EED) and the Health Technology Assessment (HTA) Database were searched, accessed through the Cochrane library interface (http://onlinelibrary.wiley.com/cochranelibrary/search8).The search strategies included Medical Subject Heading (MeSH) terms and text words.Each follows a similar structure: economic evaluation-related terms AND WDM-related terms AND limitation terms about human studies and English language.The original searches and two update searches were conducted on 21 July 2020,18 th July 2022, and 23 rd July 2023 respectively.No restriction by publication year was applied.The full search strategy is reported in S1 Text.The reference lists of all identified WDMs were hand searched for further relevant studies which may have been missed by the electronic searches.Retrieved search results were downloaded into Endnote reference management software (Clarivate Analytics, version X9.3.3).

Assessment of abstracts and papers for inclusion
Screening of abstracts and papers against the inclusion criteria was undertaken independently by two reviewers (HJ and XL).Final inclusion of studies in the review was determined by agreement of both reviewers, with disagreements resolved by discussion.

Data extraction and analysis
Data were extracted by one reviewer (HJ) and checked by a second reviewer (XL), with disagreements resolved by discussion.The following information was extracted using Microsoft Excel (Microsoft Corporation, Microsoft 365 subscription) from all included studies: author; year; country; disease area; whether the model met the WDM or pathway model criteria by demonstration or authors' reporting; number of decision points addressed; type of economic evaluation; main effectiveness outcome; modelling method; software; economic perspective; decision rule(s) used; affiliation of the corresponding author; and other information relevant for quality assessment.Study characteristics were summarised descriptively.

Quality assessment
Four commonly used checklists for economic evaluations were considered for the quality assessment of the current review, including the BMJ checklist (or Drummond checklist) [34], the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement [35], Philips's checklist [36], and the NICE checklist [37].Of these four checklists, the NICE checklist [37] was deemed to be most appropriate for the current review because (i) it focuses on the methodological quality of studies, as opposed to reporting quality (e.g. the CHEERS statement [35] focuses on the reporting quality rather than the methodological quality); (ii) it is appropriate for modelling studies, as opposed to trial-based economic evaluation (e.g. the BMJ checklist [34] is more appropriate for trial-based economic evaluation); and (iii) it allows the users to make an overall judgement regarding the methodological quality of the studies assessed.Therefore, it is easier to summarise and compare the methodological quality of a large number of included studies using the NICE checklist, compared to those checklists which do not provide an overall judgement regarding the methodological quality of the studies assessed (e.g.Philips's checklist [36]).
The NICE checklist consists of two sections: Section 1 aims to assess the applicability of a study to the decision problems that need to be addressed, whilst Section 2 aims to assess the methodological quality of the study.Given that the aim of the review was to assess the methodological quality of the included study, rather than to assess the applicability of the model results to the UK setting, only Section 2 of the NICE checklist was used.Section 2 consists of twelve quality criteria and an overall assessment.Based on the number and importance of quality criteria that a study fails, an assessment regarding the overall methodological quality of the study can be classified into one of the following categories: (i) very serious limitations-the study fails to meet one or more quality criteria, and this is highly likely to change the conclusions about cost-effectiveness; (ii) potentially serious limitations-the study fails to meet one or more quality criteria, and this could change the conclusions about cost-effectiveness; and (iii) minor limitations-the study meets all quality criteria, or fails to meet one or more quality criteria but this is unlikely to change the conclusions about cost-effectiveness.
Two reviewers (HJ and XL) performed quality assessment for all included studies, with disagreements resolved by discussion.

Study identification and selection
The detailed results of the literature search are reported in S1 Text.A modified preferred reporting items for systematic reviews and meta-analyses (PRISMA) diagram for the literature selection process is provided in Fig 1 .A total of 7,090 citations were retrieved, with an additional 36 citations identified from checking the references of included studies.After removing duplicates, 4,803 citations remained.Of the 4,803 abstracts reviewed, 4,602 were excluded for clearly failing to meet at least one inclusion criterion or meeting at least one exclusion criterion, leaving 201 for full-text review.Of these, 41 were published only as abstracts and were excluded.Full texts of the remaining 160 citations were retrieved for detailed review.Of these, 40 papers reporting 44 WDMs satisfied the predefined inclusion criteria and were included in the review.The inter-reviewer agreement between HJ and XL, measured by Cohen's kappa was 0.58, which indicates moderate agreement.A list of excluded studies with reasons are reported in S1 Table .A list of identified pathways models is reported in S2 Table.
The complete evidence table for all included WDMs is reported in S3 Table.

Quality assessment
The results of the quality assessment for WDMs are summarised below; further detail is provided in the S4  According to the quality assessment results of the NICE checklist, of all 44 WDMs, 35 were deemed to have very serious limitations (79.5%), including all PopMoD models; five were deemed to have potentially serious limitations (11.4%) [47,54,55,59,71]; and four were deemed to have minor limitations (9.1%) [30,31,61,65].The performance of included Pop-Mod WDMs (n = 11), other WDMs (n = 33), and all WDMs (n = 44) on all items of the NICE checklist is shown in Fig 2A -2C, respectively.
Of the PopMoD models, all of which were deemed to have very serious limitations, key problems identified included (Fig 2A 3. Insufficient sensitivity analyses, for example, deterministic analysis for less than three parameters, or lack of probabilistic sensitivity analysis (PSA): PSA is useful for assessing the impact of joint uncertainty of multiple parameters simultaneously.However, the current version of PopMod does not support PSA and of the eleven PopMod WDMs included, only four of them conducted PSA [39,41,74,75], and this required the use of an additional software package (MCLeague).
Of the 33 other WDMs, 24 were deemed to have very serious limitations (72.7%), five were deemed to have potentially serious limitations (15.2%), and four were deemed to have minor limitations (12.1%).Four key problems were identified (Fig 2B).First, twenty-nine WDMs (87.9%) failed to include all important and relevant outcomes and costs, for example, dis-utilities and costs resulting from AEs of interventions.Second, twenty-two WDMs (66.7%) failed to explore important uncertainties in their analysis using PSA or only conducted one-way sensitivity analyses for less than three parameters.Third, twenty WDMs (60.6%) did not report the source of resource use data or did not obtain resource use data from the best available source.For example, resource use data were estimated based on expert opinion or obtained from countries other than the one(s) of interest.Fourth, eighteen WDMs (54.5%) did not report their source of unit cost data or did not obtain unit cost data from the best available source.For example, unit cost data were estimated based on expert opinion or obtained from countries other than the one(s) of interest.

Main findings and interpretation
Our review identified 44 WDMs.The first WDM was published as early as 1977 [29].Over 70% of WDMs were published after 2010.The main disease areas covered by existing WDMs are heart disease, cancer, AIDS and metabolic disease, all of which are associated with significant disease burden.Only three WDMs (8%) were developed by commercial organisations [26,45,52].This might be because most commercial companies are more likely to be motivated to develop piecewise models which evaluate a specific (type of) intervention, rather than developing WDMs which covers all interventions across the entire care pathway.
The majority of WDMs were of poor quality which means they may require significant modification before they can be re-used.These limitations included failure to consider the disutilities and cost caused by AEs of interventions, and lack of PSA.It was estimated that in OECD countries, 15% of total hospital activity and expenditure was a direct result of AEs [76].Worldwide, the total disability-adjusted life years (DALYs) due to AEs of medical treatment was estimated to be 62.8 per 100,000 population in 2017 [77].It has been reported that, for patients with schizophrenia [25], the cost-effectiveness of an intervention tends to be driven by its AE profile, rather than its clinical effectiveness.Given the potentially substantial impact of adverse events on cost-effectiveness results, many checklists and guidance for health economic analysis, such as the NICE checklist [37], the Philips's checklist [36], Cooper's hierarchy [78] and the ISPOR good practice guidance for budget impact analysis [79], all recommend that the dis-utilities and/or cost caused by AEs of interventions need to be considered in health economic analyses.PSA is mandated by many health technology assessment agencies, including the UK's National Institute for Health and Care Excellence (NICE) [1] and the Canadian Agency for Drugs and Technologies in Health (CADTH) [80].PSA is also recommended in the guidelines produced by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [81].This is because for non-linear decision models (i.e. the relationship between the input data and the outcomes is not linear), PSA is required to provide a reliable estimate of mean costs and outcomes [1].Given the complexity of entire disease pathways, WDMs are likely to be non-linear models.Therefore, being able to run PSA is an important function of a WDM.
Other limitations which may reduce the possibility or value of the WDMs for future use included (i) no reporting of whether their models can be accessed by other researchers or not; (ii) choice of modelling method and (iii) choice of modelling software.Of all WDMs included, only 23% used an individual-level modelling method.In order to evaluate multiple decision questions across the entire care pathway, a WDM often needs to incorporate a large number of events of interest and the relevant risk factors for all events of interest.If a cohort-level model was used to represent a large number of events and risk factors, thousands or even millions of health states will need to be built, which may place a huge burden on the model development and validation [82,83].Compared with cohort-level models, individual-level models might be more appropriate for developing a WDM, because they allow individual patient characteristics (e.g., risk factors for events of interest) and histories (e.g., events of interest) to be recorded [83][84][85].In addition, individual-level models allow patients' risk of events to change over time, depending on patient characteristics and previous events.Therefore, the methodological framework for developing WDMs [9] suggest that 'given the level of depth and flexibility required to transfer the decision node across the model pathway, it is highly likely that individual-level simulation will be required."One drawback of individual-level models is they require more computation time to run PSA compared with cohort-level models [86][87][88].However, as computing power increases over time, the difference in computational burden between cohort-level and individual-level models reduces significantly [82,83].
In terms of modelling software, our review found that a quarter of WDMs were developed using PopMoD.The main benefits of using PopMod are that such models are easy to build, free to use, allow modelling of one comorbidity, and support separate modelling of age and time [38].In addition, the use of a standard modelling template facilitates comparison of results across different disease areas.However, PopMoD may not be the ideal tool in the context of building a WDM.Due to its simple structure with only four health states, PopMod does not allow modelling of disease relapse or progression or modelling of AEs, both of which are important considerations for a WDM which covers the entire natural disease history and a range of interventions.Other WDMs included in the review were developed using various modelling software, most of which are proprietary specialist software, such as Simul8, TreeAge and Matlab.Compared with open-source coding languages such as R and Python, specialist software might be easier to learn and use as they are equipped with pre-defined modules/functions.However, these software packages are often associated with a license fee.

Use of WDMs in clinical guidelines
By covering the entire care pathway and most key interventions within a single modelling framework, whole disease modelling can be used as an ideal foundation for economic evaluations in clinical guidelines.So far, whole disease modelling has been piloted on three published NICE clinical guidelines, covering three different disease areas: colorectal cancer [89], prostate cancer [90] and atrial fibrillation [91].The reported advantages of using whole disease modelling for the selected NICE guidelines are: (a) it produces a considerably larger amount of economic information compared to traditional 'piecewise' models; (b) it improves consistency of cost-effectiveness results by using a single analytic framework and a common set of assumptions and data sources; (c) it can be used to compare the cost-effectiveness of interventions at different parts of the pathway (e.g.prevention and treatment), and explore system interdependencies between different interventions; and (d) once developed, WDMs can be reused to consider other related questions or to incorporate new evidence [30,32].
The reported disadvantages of using whole disease modelling are mainly related to the technical and practical barriers.Large and complex models like WDMs are likely to be more prone to verification (programming) errors, more difficult to validate, and more difficult to explain to decision-makers than simple models.As a result, the development of each WDM has been estimated to take at least 12 months' time of a full-time modeller.However, as Professor Alan Williams [92] pointed out, if creating such large-scale models is horrendously complicated, it is because reality is horrendously complicated; the more complex the reality is, the more dangerous it is to rely on intuitive short-cuts rather than careful analyses.Therefore, 'more complex areas require models that respect complexity' [84].Given that WDMs, once developed, can be adapted and reused to provide ongoing support for decision-making across the entire care pathway, in the long-term, the benefits of using whole disease modelling in the context of clinical guidelines may outweigh the initial investment of time and money.

Recommendations for future research
Based on the findings of this review, there are a number of recommendations for future research: 1.The reporting of future WDMs should follow the CHEERS statement [35].In addition to the items listed in the CHEERS statement, we recommend that the following information should also be reported to help other researchers to decide whether the WDM can be reused/adapted to their settings: (1) software used to develop the model; and (2) a statement about access to the model, e.g.whether their model can be accessed by other researchers, and under what conditions.It is also recommended that a user manual is developed which includes a detailed description of each component of the model structure, all input data and instructions on how to use and/or adapt the model, as recommended by the STRESS guidelines for reporting models [93].
2. It is recommended that for future WDMs, the appropriateness of alternative modelling methods should be assessed and the chosen method justified, this can be achieved by the application of model selection tools, such as the revised Brennan et al taxonomy [82].
3. The health and cost impacts of important AEs of interventions should be considered for inclusion in the model and only excluded where these impacts are negligible and a clear justification is provided.
4. Extensive sensitivity analyses need to be conducted to explore all key uncertainties in the model.As a minimum, one-way sensitivity analysis for all key parameters and PSA should be conducted.
In addition to the recommendation listed above, future research exploring methods for reducing the development time of WDMs is required to encourage greater use of the whole disease modelling approach.There are two solutions which could potentially help to reduce the development time of a WDM.The first solution is to adopt a team approach.The development of a WDM can be divided into several interrelated but different task modules, each of which requires a different set of skills (e.g., development of a conceptual model, identification and preparation of relevant input data, implementing the conceptual model within a computer software, and communicating the model results to stakeholders).This means the development work for a WDM can be potentially assigned to more than one researcher working in parallel, which would accelerate the development time of a WDM.However, the adoption of a team approach will lead to increased labour costs.The second solution is to develop templates for whole disease modelling.There are some similarities in terms of building health economic models for diseases of the same or similar types.For example, the modelling of cancer usually involves tumour progression from early stages to late stages, while the modelling of mental health problems usually involves repeated transitions between remission and relapse.Even for patients with different types of diseases, their QALYs accumulated at a specific stage of the care pathway usually depend on similar factors, such as: patient's starting age, end age, current disease status, comorbidities and/or adverse events of interventions.The code for implementing these common functions can be made into a modelling template (or different templates for different types of disease) and published online.These templates may help to reduce the development time of future WDMs.

Strengths & limitations
Strengths.To our knowledge, this is the first systematic review which outlines the availability and quality of WDMs for any disease areas.The information reported by this systematic review can be used to help researchers, commissioners or other stakeholders to rapidly locate relevant WDMs in the disease areas that they are interested in and critically appraise existing WDMs.Recommendations for future research can be used to fill evidence gaps and improve the quality and reporting of future WDMs.
Limitations.This review is subject to at least five main limitations.Firstly, our search might not have identified all models which meet the criteria for a WDM.When designing the search strategy, we tried to include any search terms which might be relevant to a WDM, including terms such as "(full/comprehensive/entire/whole) adj3 (pathway*/system*/guideline*/disease*)", "upstream adj2 downstream", and "prevent* adj7 treat*".However, unless we searched for all models for any diseases, there is no guarantee that all models meeting the criteria of a WDM have been identified.In addition, non-English literature and grey literature were not searched due to constraints in time, resources, and expertise within the reviewing team for non-English languages; and the inter-reviewer agreement is moderate (Cohen's kappa = 0.58).The two reviewers (HJ and XL) tried to be more inclusive rather than exclusive when discussing any disagreed studies, however, there is still a possibility that some relevant WDMs have been missed by our review.Since the aim of this review was to provide an overview of existing WDMs, rather than to synthesise the results of the identified studies, we reckon the negative consequences of missing relevant studies are relatively small.Secondly, it should be noted that the definition of a WDM used for this review is more inclusive than the original definition provided in Tappenden et al. [9].For example, Tappenden et al. defines a WDM as a model which includes the entire preclinical and post-diagnostic pathways for a given disease, allows the use of disease-level constrained optimisation, and allows decision node to be transferred across the modelled pathway.Since models meeting such criteria are rare, we decided to relax the definition of a WDM for this review to include models which can evaluate multiple decision points covering both the prevention and treatment of the disease simultaneously.This means not all WDMs included in this review cover the entire preclinical and post-diagnostic pathways for a given disease.As a result, not all included WDMs allow the use of disease-level constrained optimisation-of the 43 WDMs included, only four [31,[49][50][51] demonstrated that they allow the use of disease-level constrained optimisation.In addition, those included WDMs which were developed using a cohort-level modelling method are unlikely to allow a decision node to be transferred across the modelled pathway.Thirdly, whether a paper meets our inclusion criteria or not was determined based on the content of each individual paper, rather than the content of a series of related papers.Therefore, we might have missed those models which were used to address multiple decision points in a series of papers, but each paper only used that model to address one or two decision points.Fourthly, for those models which claimed they can be used to address three or more decision points but did not demonstrate this in the paper, whether they met the inclusion criteria of a WDM/pathway model or not was based on the authors' reporting rather than our own assessment.Finally, we used the NICE checklist for economic evaluations [37] for assessing the quality of included WDMs.However, it should be noted that not all important aspects of quality assessment, such as model performance (e.g.comparing model results to real-world results) or model validation activities, were covered by the NICE checklist.In addition, the importance of quality criteria that a study fails (i.e.how likely this will change the conclusions about costeffectiveness) was based on the reviewer's judgement.Therefore, our results of quality assessment need to be interpreted with caution.

Conclusion
Despite their significant resource requirements associated with model development, there has been a significant increase in the number of WDMs since 2010.The main disease areas covered by existing WDMs are heart disease, cancer, metabolic disease and AIDS.A quarter of included WDMs were multi-state dynamic life table models developed using PopMoD; the remaining WDMs were developed using various modelling methods and software.The majority of WDMs were of poor quality which means they may require significant modification before they can be re-used, such as modelling AEs of interventions and incorporation of PSA.It is recommended that sufficient details of the WDMs need to be reported to allow future reuse/adaptation.

Table 2 .
[30,31,50,51] model addressed less than two decision points in the paper can still be considered to a WDM if the authors clearly reported their model can be used to address more than 2 decision points.c.Twelve studies covered more than one country.d.Within a cost-utility analysis, effectiveness is measured either as QALYs or DALYs.e.Four studies supported more than one decision rule[30,31,50,51].https://doi.org/10.1371/journal.pone.0291366.t002(10/44, 22.7%) did not report the evidence sources used to inform model parameters relating to clinical effectiveness of the technologies used within the treatment pathways. b ): 1. Inadequate model structure (11/11, 100%): PopMod is aimed at modelling initial disease treatment only and does not consider relapse or progress status; therefore, it is unlikely to capture the complexity of natural disease history or the entire care pathway.2. Failure to include all important and relevant outcomes and costs (11/11, 100%): the standard PopMod model only has four health states, including two groups with specific disease conditions, a group with the combined condition and a group with neither condition; as a result, PopMod has very limited capacity in modelling the health impacts of any adverse events (AEs) of interventions.None of the eleven PopMod WDMs in this review modelled any treatment-related AEs or justified their exclusion.